non-square VAE tiling #3
Conversation
|
Overall nice work, I like the way you refactored the |
wbruna
force-pushed
the
vae_tiling_non_square
branch
from
7ab3d61 to
680140d
Compare
|
I've fixed the latent/image confusion, and added support for two floating point factors for the env var. About the factor: I don't think it makes sense to have more than 1 tile across a dimension, so I'm considering a factor below 1 as meaning "fraction of the latent dimension", and above 1 "how many tiles across the latent dimension", so we can try out both to see which one works best. |
That's not taking the overlap into account, right? |
No, just a plain division right now. |
|
The way to compute the tile size so there are only |
|
Hm... Just let me check if I'm understanding how that overlapping factor works. We want N tiles, of size T, to fill the length L. To avoid boundary issues, we start by placing a first tile T. If the overlapping factor means "which fraction of the tile is overlapping with its neighbor", each of the additional N-1 tiles will overlap T · overlap with the already placed tile, thus covering an additional area of T · (1-overlap). So we have: For a 64 side with 3 tiles and overlap 0.5, this gives me a pretty reasonable tile size 32. Am I on the right track here? 🙂 |
|
Yes that seems right. I'm not sure how I ended up with the |
|
It's mostly working. I had to request slightly different sizes, depending on the overlapping factor; for instance, for a 768x768 image with overlapping 0.5, a 4.0 factor gives me 4 tiles across; lowering the overlapping to 0.3, I get 5; but 3.9 gives me the expected 4... so it could be just rounding. |
Indeed, adding std::round calls when multiplying by the factors seems to avoid that weirdness. |
| if ((overshoot_dim != non_tile_overlap) && (overshoot_dim <= num_tiles_dim * (tile_size / 2 - tile_overlap))) { | ||
| // if tiles don't fit perfectly using the desired overlap | ||
| // and there is enough room to squeeze an extra tile without overlap becoming >0.5 | ||
| num_tiles_dim++; | ||
| } |
There was a problem hiding this comment.
@wbruna I think it's actually because of this.
If I remember correctly I added this to make sure the overlap was preferably bigger rather than smaller than the target (because less overlap tend to cause more noticable transitions).
There was a problem hiding this comment.
If your fix with rounding doesn't work, removing these would work, though I think it's preferable to keep it.
There was a problem hiding this comment.
Seems to be working fine so far, no matter the overlap.
|
All good now i think |
* refactor tile number calculation * support non-square tiles * add env var to change tile overlap * add safeguards and better error messages for SD_TILE_OVERLAP * add safeguards and include overlapping factor for SD_TILE_SIZE * avoid rounding issues when specifying SD_TILE_SIZE as a factor * lower SD_TILE_OVERLAP limit * zero-init empty output buffer
Just an experiment to give us even more parameters to play with 🙂
I noticed sd_tiling didn't have anything forcing the tiles to be square, so I've extended SD_TILE_SIZE to support specifying tile sizes directly (like "32x48"), or as a floating-point factor applied to the image dimensions.
The factor thing is interesting, because it can be used to keep the number of tiles in both directions more or less constant, which makes a specific value fast for many different image ratios. For instance, around 15.2 (weird number because I just apply it as-is right now) is just enough to keep it at 4 iterations, which on my card is consistently faster than non-tiled VAE. In hindsight, it probably makes more sense to specify it directly as a number of tiles in each direction, though.